\section{Applying the Lifted Transformation \gmtoautosarlifted}
\label{s:applyingLifted}
\label{s:applying}

The aim of this case study is to investigate the feasibility of applying
industrial-grade transformations to product lines via lifting~\cite{salay14}.
 We thus lifted \gmtoautosar and applied
it to various input product lines with the goal to answer the following research
questions:

\begin{enumerate}[RQ1:]
\item Does \gmtoautosarlifted scale to industrial-sized SPLs?
\item How sensitive is it to the complexity of the product line?
\end{enumerate}

To answer RQ1, we generated realistic product lines, based on input from our
industrial partners. We then applied \gmtoautosarlifted to them and measured two
variables:
(a) total runtime, and (b) complexity of presence conditions of the output.
We used the clause-to-variable ratio as a measure of the complexity of presence
conditions because it is a well-known metric for evaluating the complexity of
queries to SAT solvers.
% \begin{inparaenum}[(a)]
% \item runtime, 
% \item increase in the size of presence conditions
% , as measured by comparing
% the average clause-to-variable
% \mc{this is not a normal concept - it needs additional explanation, and, in
% fact, I propose postponing the definition of the size of presence conditions to
% a bit later}
% \mf{Unsure what you propose to do exactly.}
% ratio of the input and output product lines of
% \gmtoautosarlifted.
% \end{inparaenum}
To answer RQ2, we varied the size of the generated product lines in terms of the
size of the domain model and the number of features in the feature model.

\subsubsection{Setup.}
Due to limitations of publication of sensitive industrial data, we opted to use
a {\em realistic} rather than {\em real} product lines, 
constructed as follows:  

\noindent{\bf 1)}
Using publicly available examples~\cite{simulinkEGS}, we created the exemplar
product line described in Sec.~\ref{s:exemplar}.  As described earlier, its
domain model consists of 201 elements and its feature model has 6 features.
50\% of domain model elements in
the model had a single feature presence condition, whereas the presence
conditions of the other 50\% consisted of conjunctive clauses of 2-3 features.
The overall product line was validated with input from our industrial partners.

\noindent{\bf 2)}
We consulted our industrial partners regarding the characteristics of a
typical product line. We were given the following parameters for a typical
product line of DOORS requirements: 
\begin{inparaenum}[(a)]
\item domain model size is 400 elements, 
\item the number of feature variables is 25,
\item 1/8th of elements are variation points,
\item an average clause-to-variable ratio of the presence conditions is
$^2/_{25}=0.08$, i.e. an average presence condition consists of 2 clauses
containing any of the 25 feature variables. 
\end{inparaenum}

\noindent{\bf 3)}
We used the exemplar model built in step 1 as a seed to create product lines of
varying sizes for the model and the set of features, i.e., varying parameters (a) and (b)
from step 2 while keeping parameters (c) and (d) constant.
Therefore, models of increasing sizes were obtained by cloning the
exemplar domain model to create models of 200, 400, 800, 1600 and 3200 elements. 
To obtain product lines with different numbers of feature variables, we cloned
the feature model of the exemplar, creating feature models with 6, 12, 24, 48,
and 96 features.
The product line with 400 elements and 24 features corresponds to the parameters
reported by our industrial partners in the previous step. 
Each variation point was assigned a randomly generated presence condition based
on the presence conditions of the exemplar.

\begin{figure}[t]
\centering
\subfloat[][]{\includegraphics[width=0.5\textwidth]{imgs/timeLifting.pdf}} \hfill
\subfloat[][]{\includegraphics[width=0.5\textwidth]{imgs/complexityLifting.pdf}}
\caption{(a) Observed increase in running time. (b) Observed increase in the
size of presence conditions.} 
\label{f:results}
%\vspace{-0.2in}
\end{figure}

We executed the experiments on a computer with Intel Core i7-2600
3.40GHz $\times$4 cores (8 logical) and 8GB RAM, running Ubuntu-64.

\subsubsection{Results.}
Fig.~\ref{f:results}(a) shows the observed runtimes of applying
\gmtoautosarlifted to product lines with domain models of increasing size. One
line is plotted for each feature set size.  For comparison, we also
include the runtime of applying \gmtoautosar to models (not product lines) of
different sizes.
% shows the observed effect of model size to the
% increase of runtime for different size categories of product lines, comparing it
% to the non-lifted case (i.e., applying \gmtoautosar a model).
% {\bf MC:  I cannot parse this sentence~}
Fig.~\ref{f:results}(b) shows the clause-to-variable ratio of output
product lines for inputs of varying size of domain model. One line is plotted
for each feature set size. For comparison, we also include the
clause-to-variable ratio of the input product line.
% to the size of the presence conditions of the output, measured using the average
% clause-to-variable ratio. {\bf MC:  Again, trouble parsing.}
% The outputs are compared to the constant
% clause-to-variable ratio of the input product lines. 

With respect to RQ2, we note that runtime grows exponentially with the size of
the domain model, while product lines with larger feature sets take longer to
transform.  The size of presence conditions also  grows exponentially with
increasing domain model sizes, and is two to three orders of magnitude larger
than the input. Applying \gmtoautosarlifted to product lines with smaller size
of the feature set results in a larger increase to the clause-to-variable
ratio.
With regard to the sensitivity of \gmtoautosarlifted to size of the domain
model, we observe that runtime follows the expected pattern of exponential
increase. Since the non-lifted version also grows exponentially, we conclude
that this exponential increase is not solely due to the use of a SAT solver but
also due to the inherent complexity of graph-rewriting-based model
transformations.
With regard to the sensitivity of \gmtoautosarlifted to the size of presence
conditions, we again observe an expected pattern of exponential increase.
However, the increase is orders of magnitude large which is explained by the
fact that our current implementation of \gmtoautosarlifted does
not perform any propositional simplification.

With respect to RQ1, we observe that for sizes of domain model and feature set
that correspond to the description of real GM product lines, the observed
runtime of \gmtoautosarlifted is 3.59 seconds, compared to 3.25 for
\gmtoautosar. These differences in runtime indicate that \gmtoautosarlifted
scales well in terms of runtime. On the other hand we observe that
the clause-to-variable ratio increased from 0.08 to 293.53, meaning that the
output presence conditions contained a very large number of clauses.
This points to the need to further optimize the DSLTrans engine, taking care to
strike a balance between runtime and propositional simplification.
Additionally, we note that the observed clause-to-variable ratio is not
close to 4.26, which is considered to be the hardest for automated SAT
solving~\cite{randomsat}.  

\subsubsection{Threats to Validity.}
There are two main threats to validity: First, the seed model was constructed
using non-GM data, but rather publicly available automotive examples.  Second,
product lines of different sizes of domain model and feature set were
artificially constructed by cloning the seed model.  Both these issues stem from
the fact that we could not access to real product lines due to limitations to
publication of sensitive industrial data.  
To mitigate the first concern, we asked industrial partners to validate that our
exemplar is realistic in terms of structure and size. To mitigate the second
concern, we ensured that our cloning process resulted in product lines that had
characteristics that were consistent with the parameters given by our
industrial partners (number of variation points, average clause-to-variable
ratio, shape of the presence conditions).




